Statistical machine translation employing efficient parameter training

ABSTRACT

A statistical machine translation (SMT) system employs a conditional translation probability conditioned on the source language content. A model parameters optimization engine is configured to optimize values of parameters of the conditional translation probability using a translation pool comprising candidate aligned translations for source language sentences having reference translations. The model parameters optimization engine adds candidate aligned translations to the translation pool by sampling available candidate aligned translations in accordance with the conditional translation probability.

BACKGROUND

The following relates to the machine translation arts, the statisticalmachine translation arts, and so forth.

Machine (or automated) translation from a source language to a targetlanguage is known. For example, such machine translation mayautomatically translate a source-language sentence in English, French,Chinese, or another natural language, to a corresponding target-languagesentence in another natural language. Some machine translation systemsfurther include a user interface via which the machine translation ispresented to a user as a proposed translation, which may be accepted,rejected, or modified by the user via the user interface.

In translation memory systems, a translation memory stores previouslytranslated text as source language content and corresponding translatedtarget language content, with corresponding textual units (e.g., words,phrases, sentences, or so forth) in the source and target languagesassociated together. When source-language content is received fortranslation, it is compared with the source-language contents of thetranslation memory. If a match, or approximate match, is found, thecorresponding aligned target language content is presented to the user.If the match is approximate, the user may also be informed of thedifferences. The translation memory approach depends upon the memorycontents being accurate and sufficiently comprehensive to encompass ausefully large portion of the source-language content received fortranslation.

Another known technique for machine translation is statistical machinetranslation (SMT). In this approach, a database ofsource-language/target language phrases are stored as a phrase table.(The term “phrase” as used herein and in the SMT literature generally isto be understood as a unit of text, e.g. a word or sequence of words, insome instances possibly including punctuation—the term “phrase” is notlimited herein or in the SMT literature generally to grammaticalphrases.) A translation model is provided or developed. This modelcomprises an aligned translation conditional probability. The “alignedtranslation” comprises one or more target language phrases in aparticular sequence (i.e., alignment), with each target language phrasecorresponding to a phrase of the source language content. In operation,the SMT generates candidate translations for received source languagecontent to be translated by selecting target language phrases from thephrase table that match source language phrases of the source languagecontent. The translation model is used to assess the candidatetranslations so as to select a translation having a high probability asassessed by the model. Since the number of candidate translations can betoo large to exhaustively search, in some SMT configurations thetranslation model is used to guide the generation of candidatetranslations, for example by modifying a previously generated candidatetranslations to generate new candidate translations having highprobabilities as assessed by the model.

Similarly to the translation memory approach, SMT depends on thecomprehensiveness and accuracy of the phrase table. However, since thephrases are generally substantially shorter that textual units of atranslation memory, it is generally easier to generate an accurate andreasonably comprehensive phrase table. SMT also depends on the accuracyof the translation model. Toward this end, the translation model isgenerally constructed to be “tunable”, that is, the translation modelincludes model parameters that can be optimized based on a developmentdataset comprising source language sentences and corresponding alignedtarget language translations.

The following discloses various improvements in machine translationapparatuses and methods.

BRIEF DESCRIPTION

In some illustrative embodiments disclosed as illustrative examplesherein, a method comprises: translating source language content in asource natural language to a target natural language using statisticalmachine translation (SMT) employing a conditional translationprobability conditioned on the source language content; and optimizingvalues of parameters of the conditional translation probability by aniterative optimization process operating on a translation pool, theoptimizing including adding candidate aligned translations to thetranslation pool by sampling available candidate aligned translationsfor a source language sentence in accordance with the conditionaltranslation probability. The SMT and the optimizing are suitablyimplemented by a SMT system embodied by at least one digital processor.

In some illustrative embodiments disclosed as illustrative examplesherein, a statistical machine translation (SMT) system is embodied by atleast one digital processor performing the method of the immediatelypreceding paragraph. In some illustrative embodiments disclosed asillustrative examples herein, a storage medium stores instructionsexecutable on a digital processor to perform the method of theimmediately preceding paragraph.

In some illustrative embodiments disclosed as illustrative examplesherein, a storage medium stores instructions executable on a digitalprocessor to perform a method including (i) translating source languagecontent in a source natural language to a target natural language basedon a conditional translation probability conditioned on the sourcelanguage content and (ii) tuning the conditional translation probabilityusing a translation pool, wherein the tuning includes (I) selectingcandidate aligned translations for a source language sentence having areference translation by sampling available candidate alignedtranslations for the source language sentence in accordance with theconditional translation probability conditioned on the source languagesentence, and (II) adding the selected candidate aligned translations tothe translation pool.

In some illustrative embodiments disclosed as illustrative examplesherein, an apparatus comprises: a statistical machine translation (SMT)system employing a conditional translation probability conditioned onthe source language content; and a model parameters optimization engineconfigured to optimize values of parameters of the conditionaltranslation probability using a translation pool comprising candidatealigned translations for source language sentences having referencetranslations, the model parameters optimization engine adding candidatealigned translations to the translation pool by sampling availablecandidate aligned translations in accordance with the conditionaltranslation probability. The SMT system and the model parametersoptimization engine are suitably embodied by one or more digitalprocessors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 diagrammatically shows a statistical machine translation (SMT)system including a model parameter optimization engine as set forthherein.

FIG. 2 diagrammatically shows an illustrative translation latticerepresenting some possible target French-language translations of thesource English-language sentence “I have a blue car.”

FIG. 3 diagrammatically shows reachable translations for a singlesentence using a translation model having two feature functions.

FIG. 4 diagrammatically illustrates operation of the model parameteroptimization engine of the SMT system of FIG. 1.

DETAILED DESCRIPTION

With reference to FIG. 1, a statistical machine translation (SMT) system10 is suitably implemented by one or more computers, such as anillustrative computer 12 having a display 14 or other user interfacingoutput device and a keyboard 16, mouse (not shown), or other userinterfacing input device. The one or more computers include at least onedigital processor (not illustrated) configured by software, firmware, ora combination thereof to perform statistical machine translation, thatis, configured to embody the SMT system 10. More generally, the SMTsystem 10 is embodied by one or more digital processors, which may ingeneral be processors of computers, processors of handheld digitaldevices, or one or more other suitably programmed microprocessors,microcontrollers, or so forth. The user interfacing components 12, 14are used by a human user to formulate a translation request comprisingcontent 20 expressed in a source natural language. The SMT system 10generates a translation 22 of the content 20 in a target naturallanguage, which is then displayed on the display 14 or otherwiseutilized.

It is also contemplated to omit a human user from the processing. Forexample, the source language content 20 may be generated automatically,for example by an application program that extracts the source languagecontent 20 from a document. Similarly, the target language translation22 may be utilized automatically, for example by the application programin constructing a translated version of the document in the targetlanguage.

For embodiments in which the SMT system 10 is embodied by one or moredigital processors of one or more computers, it is to be understood thatthe one or more computers may include one or more desktop computers, oneor more notebook computers, one or more network servers, one or moreInternet servers, or various combinations thereof.

In some embodiments, the one or more computers include auser-interfacing computer including the user interfacing components 14,16, and a separate network server computer having one or more digitalprocessors configured to perform the statistical machine translation. Insuch embodiments, the user interfacing computer is employed by a humanuser to formulate a translation request comprising content expressed ina source natural language, and this content is communicated via thenetwork (for example, a wired local area network, a wireless local areanetwork, the Internet, some combination thereof, or so forth) to theserver computer hosting the SMT system 10, and a translation of thecontent in a target natural language is then communicated back via thenetwork to the user interfacing computer where the translation isdisplayed on the display 14 or is otherwise conveyed to the human user.

In some embodiments, the one or more computers include a single computerincluding the user interfacing components 14, 16, and also including theone or more digital processors configured to perform the statisticalmachine translation. In such embodiments, the user interfacing computeris employed by a human user to formulate a translation, the included oneor more digital processors perform the statistical machine translation,and the resulting translation is displayed on the display 14 of thecomputer.

In some embodiments, a storage medium (not illustrated) storesinstructions executable on a digital processor to perform the disclosedstatistical machine translation techniques including the disclosedparameter optimization embodiments. The storage medium may include, forexample, one or more of the following: a hard disk or other magneticstorage medium; an optical disk or other optical medium; a read-onlymemory (ROM), electronically erasable programmable read-only memory(EEPROM), flash memory, random access memory (RAM) or other electronicstorage medium; a combination of two or more of the foregoing; or soforth.

With continuing reference to FIG. 1, the SMT system 10 translates thesource language content 20 in the source natural language to the targetnatural language (so as to generate the translation 22) usingstatistical machine translation (SMT) employing a translation model 30comprising a conditional translation probability conditioned on thesource language content. The translation model 30 receives sourcelanguage content and a possible translation for the received sourcelanguage content, and computes a conditional translation probability forthe translation that is conditioned on the received source languagecontent. The received source language content is denoted herein as f.The received corresponding translation whose conditional probabilityconditioned on f is to be computed is denoted as (e,a). This translationis an aligned translation, by which it is meant that the set of phrasese have a specified alignment a respective to corresponding phrases ofthe source language content f. If the source language content f is asingle phrase, then the alignment a has no significance and can bedisregarded.

The conditional translation probability 30 is parameterized, that is,includes model parameters 32 whose values can be adjusted to optimize ortune the conditional translation probability 30. By way of illustrativeexample, in the illustrative embodiments described herein theconditional translation probability 30 employs a log-linear model asfollows:

$\begin{matrix}{{{P\left( {e,\left. a \middle| f \right.} \right)} = \frac{\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e,a,f} \right)}}} \right)}{\sum\limits_{{({e^{\prime},a^{\prime}})} \in L}{\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e^{\prime},a^{\prime},f} \right)}}} \right)}}},} & (1)\end{matrix}$where f denotes the source language content, L denotes a set ofavailable candidate aligned translations, (e′,a′) denotes onetranslation belonging to the set L, (e,a) denotes a candidate alignedtranslation for which the conditional probability is computed, h_(k)( .. . ), k=1, . . . , K denotes a set of K feature functions, and λ_(k),k=1, . . . , K denotes the parameters 32 of the conditional translationprobability.

To generate candidate aligned translations, a phrase table 34 containingsource language-target language phrase pairs is used to constructpossible aligned target language translations for multi-phrase sourcelanguage content. For each source language phrase in the source languagecontent, the phrase table 34 is searched for possible correspondingtarget language phrases, which are then combined with various alignmentsto generate candidate aligned translations.

With continuing reference to FIG. 1 and with brief reference to FIG. 2,in some embodiments a decoder 36 performs translation latticeconstruction and updating to generate a translation lattice representingthe candidate aligned translations. FIG. 2 illustrates a possibletranslation lattice for the translation of the English-language content“I have a blue car” into French. The translation lattice is traversedfrom the start state (designated “0”) to the final state (designated(F)). The illustrative translation lattice of FIG. 2 represents threepossible translations, corresponding to the traversal paths: 0-1-2-F;0-3-4-F; and 0-5-4-F. More generally, however, the translation latticemay represent tens or hundreds or more possible translations for a givensource language content.

With continuing reference to FIG. 1, the decoder 36 consults the phrasetable 34 to generate the set L of available candidate alignedtranslations. The set L may include tens or hundreds or more possibletranslations for a given source language content, but is not necessarilyexhaustive—in other words, there may be still yet other possibletranslations that are not included in the set L of available candidatealigned translations. The decoder 36 selects the final proposedtranslation 22 from the set L of available candidate alignedtranslations. The decoder 36 chooses the translation 22 from the set Lof available candidate aligned translations so as to maximize theconditional translation probability 30 conditioned on the sourcelanguage content 20.

With reference again to the illustrative log-linear conditionaltranslation probability of Equation (1), it will be noticed that thedenominator of Equation (1) does not depend on the particular candidatetranslation (e,a) for which the conditional probability is beingcomputed. Thus, for the purpose of maximizing the conditionalprobability of Equation (1) respective to candidate translation (e,a),the denominator can be omitted, and the exponential in the numerator canalso be dropped. Thus, the decoder 36 suitably selects the translation22 as the optimal translation (e*,a*) as follows:

$\begin{matrix}{{\left( {e^{*},a^{*}} \right) = {\underset{e,a}{\arg\;\max}\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}^{opt}{h_{k}\left( {e,a,f} \right)}}} \right)}},} & (2)\end{matrix}$where λ_(k) ^(opt), k=1, . . . , K denotes values for the parameters 32of the conditional translation probability 30 that are optimized for thetranslation task, and (e*,a*) denotes the translation 22.

With brief reference again to FIG. 2, finding the optimal translationaccording to Equation (1) or Equation (2) is NP-complete. In a suitableapproach which is faster, a beam-search heuristic algorithm is used tosolve the problem approximately. In one such approach, the decoder 36proceeds by progressively extending translation prefixes by adding onenew phrase at a time, and correspondingly “consuming” portions of thesource sentence. Each prefix is associated with a node in a graph, andreceives a score according to the model. Whenever two prefixes havingexactly the same possible extensions are detected, the lower-scoring oneis merged into the other, thus creating a re-entrancy in the directedgraph, which has then the characteristics of a lattice (see the exampleof FIG. 2). Edges in the lattice are labeled with the phrase pair thatwas used to perform the corresponding extension, the source wordpositions that were covered in doing the extension, and thecorresponding increment in model score.

As used with reference to Equation (2), and as used generally herein,terms such as “optimize”, “minimize”, “maximize”, and so forth are to bebroadly construed as encompassing approximate or non-globaloptimization, minimization, maximization, or so forth. For example, inevaluating Equation (2) it is to be understood that the evaluation isover the set of available candidate aligned translations, which may notinclude the globally best translation. Moreover, even if the globallybest translation is included in the set of available candidate alignedtranslations, the evaluation of Equation (2) may employ a non-exhaustiveoptimization that may identify a locally best translation rather thanthe globally best translation. Similar broad construction is understoodto apply respective to optimization of the parameters 32 of theconditional translation probability.

The parameters λ_(k), k=1, . . . , K 32 determine the relativeimportance of the different feature functions h_(h)( . . . ), k=1, . . ., K in the global score of Equation (1) or Equation (2). The parameters32 are in some embodiments suitably tuned by cross-validation. Moregenerally, the parameters 32 are tuned by a model parametersoptimization engine 40 using a development dataset 42 comprising sourcelanguage sentences having reference translations in the target language,for example supplied by a human translator. Although the modelparameters optimization engine 40 is diagrammatically shown in FIG. 1 asan individual component of the SMT system 10, in operation the modelparameters optimization engine 40 interacts with other cooperating SMTsystem components such as the translation model 30 and the decoder 36.(See, e.g. FIG. 4 and related discussion herein). In operation theoptimization engine 40 searches the space of parameter vectors λ=[λ₁, .. . , λ_(K)]^(T) to identify a parameter vector λ ^(opt)=[λ₁ ^(opt), . .. , λ_(K) ^(opt)]^(T) that minimizes the error between translations ofthe source language sentences of the development dataset 42 made by thedecoder 36 employing the translation model 30 with the optimizedparameter values λ ^(opt). The optimization engine 40 operates on atranslation pool 44 containing candidate translations of the sourcelanguage content generated (in the illustrated embodiment) by thedecoder 36. In some suitable embodiments, the error is suitably measuredby an error measure such as the BLEU error metric 46 defined inaccordance with the bilingual evaluation understudy (BLEU) method. See,e.g. Papineni et al., “Bleu: a method for automatic evaluation ofmachine translation”, in ACL '02: Proceedings of the 40^(th) AnnualMeeting on Association for Computational Linguistics, pages 311-318(Philadelphia, Pa., USA, 2002), which is incorporated herein byreference in its entirety. The BLEU error metric 46 measures thegeometric mean of the precisions with which the translation produced bythe SMT recovers n-grams of different orders from the availablereference translation(s).

The optimization performed by the model parameters optimization engine40 is suitably a type of Minimum Error Rate Training (MERT). Thealgorithm starts by initializing the parameter vector λ. In embodimentsdisclosed herein, for each source sentence in the development dataset42, the translation pool 44 is initialized with a set of candidatetranslations generated by sampling available candidate alignedtranslations for the source language sentence in accordance with theconditional translation probability 30. Using the translation pool 44and the corresponding reference translations, an optimization procedureis run by the optimization engine 40 to update the parameter vector toan updated value λ′ which has reduced error for the translation pool 44as compared with the vector λ, as measured by BLEU or another suitableerror metric. Additional candidate aligned translations are then addedto the translation pool 44 by further sampling of the availablecandidate aligned translations for the source language sentences of thedevelopment dataset 42, again in accordance with the conditionaltranslation probability. The additional candidate aligned translationsare merged into the translation pool 44, and the process is repeatediteratively until a stopping criterion is met.

The process implemented by the optimization engine 40 has somesimilarity with a “Best-N” MERT algorithm such as is described in Och,“Minimum error rate training in statistical machine translation”, in ACL'03: Proceedings of the 41^(st) Annual Meeting on Association forComputational Linguistics, pages 160-167 (Sapporo, Japan, 2003), whichis incorporated herein by reference in its entirety. In the Best-N MERTapproach, the translation pool is updated in each iteration with a listof N-best scoring candidate translations according to the model usingthe current values of the model parameters. In contrast, the approach ofthe optimization engine 40 is to add candidate aligned translations tothe translation pool 44 for each source language sentence of the dataset42 by sampling available candidate aligned translations for the sourcelanguage sentence in accordance with the conditional translationprobability 30 conditioned on the source language sentence.

The Best-N procedure provides computational efficiency, based on theobservation that BLEU only depends on the translation receiving thehighest score by the translation model in the translation pool. This inturn means that, for any given source language sentence, itscontribution to BLEU changes only when the value of the parameterschange in such a way that the candidate translation ranking highestaccording to the model switches from one candidate translation toanother. This situation does not change when one considers all thesource language sentences in a development set instead of just one:while varying the X vector, the BLEU score changes only when there is achange at the top of the ranking of the alternatives for at least onesource language sentence in the set. In other words, BLEU is piece-wiseconstant in λ. Best-N MERT then proceeds in a computationally efficientmanner by performing an iterative line search by fixing each time thevalue of all components of λ except one: for such a free parameter aglobal optimum can be identified by enumerating all the points thatcause a change in BLEU. The value of the component is then fixed at themiddle of an interval with maximum BLEU, and the procedure is iterateduntil convergence. Since the error function is highly irregular, and theiterative line search is not guaranteed to converge to a global optimum,the procedure is repeated multiple times with different initializations,and the best convergence point is retained.

The Best-N MERT algorithm assumes at each iteration that the set ofcandidates with a chance to make it to the top (for some value of theparameter vector λ) is well represented in the translation pool.However, because the translation pool is formed by merging N-best lists,this assumption is easily violated in practice. Indeed, the N-best listoften contains only candidates displaying minor differences, andrepresents only a small sample of alternative possible translations,strongly biased by the current parameter setting.

Thus, the Best-N approach is computationally efficient, but convergesslowly if the initial parameter vector λ is far away from the optimalvalue λ ^(opt). Additionally, the N-best translations tend to be similarto each other, which again tends to lead to slow convergence.

The process implemented by the optimization engine 40 also has somesimilarity with another MERT algorithm that is described in Macherey etal., “Lattice-based minimum error rate training for statistical machinetranslation”, in EMNLP '08: Proceedings of the Conference on EmpiricalMethods in Natural Language Processing, pages 725-734 (Morristown, N.J.,USA, 2008), which is incorporated herein by reference in its entirety.The approach of Macherey et al. extends the MERT algorithm so as to usethe whole set of candidate translations compactly represented in thesearch lattice produced by the decoder, instead of only a N-best list ofcandidates extracted from it as in the Best-N approach. The use of thewhole set of candidate translations is achieved via an elegant butrelatively heavy dynamic programming algorithm that propagatessufficient statistics (called envelopes) throughout the whole searchgraph. The theoretical worst-case complexity of this algorithm reportedin Macherey et al. is O(|V∥E|log|E|), where V and E are the vertex setand the edge set of the lattice respectively.

The approach of Macherey et al. overcomes deficiencies of the Best-Napproach by using all available candidate translations. Thus, there issubstantially less concern about the N-best list being biased toward thecurrent value of the parameter vector λ, or of the N-best listcomprising similar translations. However, even though the approach ofMacherey et al. employs a relatively efficient algorithm, its use of allavailable candidate translations nonetheless results in slowper-iteration processing as compared with the Best-N approach. In otherwords, although the number of iterations is advantageously reduced byusing all available candidate translations, the computational complexityof each iteration is substantially increased.

A difference in the approach of the optimization engine 40 of FIG. 1from that of Macherey et al. is that the optimization engine 40 usesonly a sampling of the available candidate translations, whereasMacherey et al. use all available candidate translations.

The approach of the optimization engine 40 of FIG. 1 produces a set ofcandidate aligned translations that is more representative of thecomplete distribution than an N-best list, while avoiding thecomputational complexity of propagating envelopes throughout the latticeas per the method of Macherey et al. Computational complexity of theapproach of the optimization engine 40 of FIG. 1 is increased onlymarginally over the N-best list approach, while yielding significantimprovements in final translation quality comparable with theimprovements reported when using the whole lattice as per the method ofMacherey et al.

With reference to FIG. 3, advantages of the sampling approach of theoptimization engine 40 of FIG. 1 are further described by way ofexample. In FIG. 3, a simplified situation is shown, in which theconditional translation probability has only two feature functions h1,h2, and FIG. 3 plots all possible translations that can be generatedusing the phrases of the phrase table 34 for a single source languagesentence. These possible translations are indicated by open or closedcircle symbols in FIG. 3. The reference translation is indicated in FIG.3 by a “+” symbol. The “best reachable” translation is also identifiedin FIG. 3. This is the translation that can be constructed using thephrase table 34 that is closest to the reference translation as measuredby a suitable error or similarity measure such as BLEU.

A current value for the parameter vector λ is indicated in FIG. 3, alongwith the next iteration the parameter vector λ′_(N-best) which would begenerated by a single iteration of the N-best MERT algorithm. Thelimited scope of N-best lists (that is, limited to the N-best candidatetranslations for the current value of λ) rules out many alternativetranslations that would receive the highest score for some other valueor values of the parameter vector λ. As shown by the small dashedpolygon in FIG. 3, the scope of the N-best lists is limited to theextreme right end of the set of available translations.

The complete set of translations that can be produced using the phrasetable 34 (also called the “reachable translations” herein) for thesource sentence is represented in FIG. 3 as a set of vectors (i.e.,circle symbols) in the space spanned by the feature functions h1, h2.Not all such translations stand a chance to receive the highest scorefor any value of the parameter vector, though. Indeed, if translationsh, h′, h″ are such that h_(k)≦h_(k)′≦h_(k)″ for all feature k, thenthere is no value of λ that will give to h′ a score higher than both hand h″. The candidate translations that would rank first for some valueof the λ parameter vector are those on the convex envelope of theoverall candidate set (this is the outermost polygon in FIG. 3, drawnwith a solid line and encompassing all the available candidatetranslations). The set of candidate translations represented by thetranslation lattice is a subset enclosed in the larger dashed polygon inFIG. 3. This set of available candidate translations defined by thetranslation lattice is biased to contain translations ranking highaccording to the current values of the parameter vector λ used toproduce it, because of the pruning strategies that guide theconstruction of the translation lattice. Both the N-best list and thesampling approach of the optimization engine 40 of FIG. 1 are furthersubsets of the set of available candidate translations defined by thetranslation lattice. The N-best list is very biased towards availablecandidate translations that score high with the current values of theparameter vector λ: its convex envelope (that is, the smaller dashedpolygon) is very different from the envelope of the set of reachabletranslations (the outermost solid polygon), and is also very differentfrom the envelope of the available translations defined by thetranslation lattice (the larger dashed polygon). As seen in FIG. 3, the“best in N-best list” candidate translation is very far away from the“best in lattice” candidate translation which is the available candidatetranslation of the translation lattice that is closest to the referencetranslation (marked by the “+” symbol).

The convex envelope of a sampling of the available candidate alignedtranslations (the larger dashed polygon) for the source languagesentence, sampled from the translation lattice in accordance with theconditional translation probability, is indicated by the dotted polygonin FIG. 3. The sampling will generally be closer to the envelope of thewhole translation lattice, as compared with the envelope of the N-bestlist. In FIG. 3, the best candidate translation from the sampling islabeled “best in random sample”.

With continuing reference to FIG. 3, curves shown in FIG. 3 indicateregions of constant loss (that is, iso-BLEU score). FIG. 3 isdiagrammatic, and these iso-BLUE score lines are typically substantiallymore irregularly shaped than as depicted in FIG. 3. For the sourcesentence, then, the optimal choice of the parameters would be around λ*.Performing an optimization step based on the random sample envelopewould result in a more marked update ( λ′_(sample)) in the direction ofthe best parameter vector than if an N-best list is used ( λ′_(N-best)).While FIG. 3 shows a simplified example with only two feature functions,in practice the number of feature functions will be substantiallylarger, with between five and twenty feature functions being typical forsome SMT systems. In such cases, a substantially larger fraction of thecandidate translations will tend to lie on the convex envelope of theset, and not inside the convex hull.

In sum, the optimizing including adding candidate aligned translationsto the translation pool by sampling available candidate alignedtranslations for a source language sentence in accordance with theconditional translation probability 30, as disclosed herein, providestranslation pool additions that are substantially better (in terms oflikelihood of movement toward the reference translation) as comparedwith the N-best MERT approach, while having computational complexitycomparable with the N-best MERT approach and substantially lower thanthe “whole lattice” approach of Macherey et al.

With reference to FIG. 4, some illustrative embodiments of parameteroptimization suitably performed by the model parameters optimizationengine 40 are described. In FIG. 4, the development dataset 42 is shownas including the source-language sentences 50 and the correspondingtarget-language reference translations 52. In an operation 60, the modelparameters are initialized. Substantially any initialization procedureis suitable. In some embodiments the parameters are initialized to equalvalues, e.g. if there are K parameters then each parameter is set to aninitial value of 1/K (so as to provide the initial parameter vector λnormalized to unity). If a priori information is available suggestingbetter initial weight values, this information is suitably used in theinitialization operation 60. In an operation 62, the decoder 36 (seeFIG. 1) is used to construct translation lattices corresponding to thesource-language sentences 50. The operation 62 references the phrasetable 34 (see FIG. 1) in constructing the translation lattices.

In an operation 64, the model parameters optimization engine 40 (seeFIG. 1) samples the translation lattice for each source-languagesentence to generate candidate translations for addition to thetranslation pool 44. The sampling is in accordance with the conditionaltranslation probability 30, suitably using the current parameter valuesof the current parameter vector λ.

The first iteration of the operation 64 effectively initializes thetranslation pool 44, and so N sampled candidate translations are addedto the translation pool 44 in the initial iteration. In subsequentiterations, the operation 64 of adding candidate translations to thetranslation pool 44 by sampling in accordance with the conditionaltranslation probability entails merging the newly sampled candidatetranslations with the candidate translations already contained in thetranslation pool 44. Thus, if a newly sampled candidate translation isalready in the translation pool 44, it is not “added again” but ratherthe new sampling is discarded. In this case the number of candidatetranslations added during that iteration is less than N. In analternative approach, if a newly sampled candidate translation isalready in the translation pool 44 then another candidate translationmay be sampled so as to keep the number of added candidate translationsat N for each iteration.

An operation 68 updates the model parameters of the parameters vector λto obtain better parameter values as measured by a suitable metriccomparing the candidate translation selected by the decoder 36 (seeFIG. 1) for each source-language sentence 50 with the correspondingreference translation 52. In a suitable embodiment, the parametersupdate operation 68 employs the BLEU error metric 46, although othererror metrics are also contemplated. The update operation 68 operatessimilarly to the corresponding operation of a Best-N MERT algorithm, forexample, as described in Och, “Minimum error rate training instatistical machine translation”, in ACL '03: Proceedings of the 41^(st)Annual Meeting on Association for Computational Linguistics, pages160-167 (Morristown, N.J., USA, 2003), which is incorporated herein byreference in its entirety. However, the operation 68 operates on thetranslation pool 44 constructed by sampling of the translation latticein the operation 64, where the sampling is in accordance with theconditional translation probability 30.

At a decision operation 72, a suitable stopping criterion is employed todetermine whether further iterations should be performed. The stoppingcriterion can be based, for example, on the iteration-to-iterationimprovement in the value of the BLEU metric 46, and/or a maximum numberof iterations stopping criterion, and/or based on a threshold for thenorm of the update to the parameter vector λ, or so forth. If iteratingis to continue, flow returns to the translation lattice constructionoperation 62 to extend or update the translation lattice, and theextended or updated translation lattice is then sampled by the samplingoperation 64 to perform additional sampling using the conditionaltranslation probability 30 with parameter vector values updated by theupdate operation 68 of the previous iteration. In some embodiments (notshown), it is alternatively contemplated for flow to return for the nextiteration directly to the sampling operation 64, without first employingthe construction operation 62 to extend or update the lattice.

A suitable implementation of the sampling operation 64 is described. Inthis approach, N candidates are sampled from the translation latticeaccording to the probability distribution over paths induced by themodel 30, given the current setting of the λ parameters. The samplingproceeds from the root node of the translation lattice, corresponding toan empty translation candidate covering no words of the source-languagesentence 50, by choosing step by step the next edge to follow. Theprobability distribution for each possible follow-up is the posteriorprobability of following the edge given the path prefix derived from thetranslation lattice. In a suitable approach, the posterior probabilityis obtained via a preliminary backward sweep. Said another way, thecandidate translation is sampled in accordance with the conditionaltranslation probability 30, for example as set forth in Equation (1), bytraversing the translation lattice from its root node to its final nodewherein each transition from a current node to a next node is selectedbased on conditional translation probabilities (again, given by Equation(1)) of the available edges leading away from the current node.

Feature functions are incremental over the edges of the translationlattice. Accordingly, the non-normalized probability of a path includingm edges or transitions is given by:

$\begin{matrix}{{{P\left( {e_{1},\ldots\mspace{14mu},e_{m}} \right)} = {\exp\left( {\sum\limits_{i = 1}^{m}{\sigma\left( e_{i} \right)}} \right)}},} & (3)\end{matrix}$where:

$\begin{matrix}{{\sigma\left( e_{i} \right)} = {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( e_{i} \right)}}}} & (4)\end{matrix}$is the score of the edge e_(i). With a minor notational change the scoreσ(e_(i)) is also denoted herein as σ(n_(j,k)), where the edge e_(i) goesfrom translation lattice node n_(j) to translation lattice node n_(k).Further denoted herein as σ(n_(i)) is the score of node n_(i), that is,the logarithm of the cumulative unnormalized probability of all thepaths in the lattice that go from node n^(i) to a final node. Theunnormalized probability of selecting node n_(j) starting from n_(i) canthen be expressed recursively as follows:S(n _(j) |n _(i))≈exp(σ(n _(j))+σ(n _(i,j)))  (5).

The scores required to compute these sampling probabilities can beobtained by a backward pass in the lattice. Let P_(i) denote the set ofsuccessors of n_(i). Then the total unnormalized log-probability ofreaching a final state (i.e. with a complete translation) from n_(i) isgiven by:

$\begin{matrix}{{{\sigma\left( n_{i} \right)} = {\log\left( {\sum\limits_{n_{j} \in P_{i}}{\exp\left( {{\sigma\left( n_{j} \right)} + {\sigma\left( n_{i,j} \right)}} \right)}} \right)}},} & (6)\end{matrix}$where σ(n_(i))=0 is set if P_(i)={ }, that is, if n_(i) is the finalnode of the translation lattice. At the end of the backward sweep, σ(n₀)contains the unnormalized cumulative probability of all paths, that is,the partition function. Notice that this normalizing constant cancelsout when computing local sampling probabilities for traversed nodes inthe translation lattice.

Once the transition probability is known for each node, as per Equation(5), candidate translations are sampled by starting in the root node ofthe translation lattice and at each step randomly selecting among itssuccessors, until the final node is reached. The whole samplingprocedure is repeated as many times as the number of samples sought(e.g., N times in the case of sampling N candidate translations). Aftercollecting samples for each source sentence 50, the whole list is usedto grow the translation pool 44 by merging the list with thealready-present contents (if any) of the translation pool 44.

An analysis of time complexity of the parameter optimization of FIG. 4is now considered. For each line search in the inner loop of the MERTalgorithm, all methods considered here (i.e., Best-N, sampling as perthe model parameters optimization engine 40 of FIG. 1, or the approachof Macherey et al.) compute the projection of the convex envelope thatcan be scanned by leaving all components unchanged but one. If thetranslation pool 44 is created and updated using either N-best lists orsampling in accordance with the conditional translation probability 30,and M is the size of the translation pool 44, then computing theenvelope can be done in time O(M log M) using the SweepLine algorithm.(See Algorithm 1 in Macherey et al., “Lattice-based minimum error ratetraining for statistical machine translation”, in EMNLP '08: Proceedingsof the Conference on Empirical Methods in Natural Language Processing,pages 725-734 (Morristown, N.J., USA, 2008), which is incorporatedherein by reference in its entirety). As it is shown in Macherey et al.,the lattice method for computing the envelope is O(|V∥E|log|E|), where Vis the vertex set of the lattice, and E is its edge set. In someembodiments of the decoder 36 there is a maximum limit D to the alloweddistortion, and lattice vertices are organized in J priority queues(sometimes referred to as “stacks”) of size at most a, where J is thelength of the source sentence and a is a parameter of the decoder set bythe user. Also, there is a limit K to the maximum number of source wordsspanned by a phrase, and only up to c alternative translations for asame source phrase are kept in the phrase table 34. Under theseconditions, the number of outgoing edges E′ from each lattice vertex canbe bounded by a constant. A first way to see this is by considering thatif an hypothesis is extended with a phrase, then the extended hypothesismust end up in a stack at most K stacks to the right of the originalone. There are only aK places in these stacks, so |E′|≦aK holds. Analternative way to see this is to consider that after covering a givensource phrase, the decoder 36 will only consider as an option covering aspan of length 1≦k≦K in the source starting in a window of size 2Daround the most recently covered source word. Since there are at most calternative target phrases for each source phrase, it follows that:

$\begin{matrix}{{{E^{\prime}} \leq {\sum\limits_{k = 1}^{K}{c\left( {{2D} - k} \right)}}} = {{{cK}\left( {{2D} - \frac{K + 1}{2}} \right)}.}} & (7)\end{matrix}$

Since the number of edges leaving each node is bounded by a constant, itis |E|=Θ(|V|), and the method of Macherey et al. is O(|V|² log|V|). Themaximum number of vertices in the translation lattice is limited by thecapacity of the stacks—that is, |V|≦aJ. This leads to a complexity ofO(J² log J) for the inner loop of the method of Macherey et al.

The complexity is driven by the length of the source sentence in thecase of the method of Macherey et al., and by the size of thetranslation pool in the case of both the N-best list method and thesampling method employed by the optimization engine 40 of FIG. 1 herein.The latter two methods are asymptotically more effective as long as thesize of the sample/N-best list grows sub-quadratically in the length ofthe sentence.

The foregoing addresses the complexity of the innermost loop whichsearches for a global optimum along a line in the parameter space. Thisline search is repeated many times, and accordingly has a substantialimpact on the overall complexity of each of the parameters optimizationmethods (top N, sampling, or the method of Macherey et al.). In thefollowing, the different methods are considered in terms of theoperations that are performed as part of the outer iteration, that isupon redecoding the development set with a new parameter vector.

For the N-best list method, this outer iteration entails constructing anN-best list from the translation lattice. This can be done in timelinear in the size J of the sentence and in N with a backward sweep inthe translation lattice. In the case of the method of Macherey et al.,the outer iteration does not entail any operations at all, since thewhole lattice is passed over for envelope propagation to the inner loop.

The sampling method implemented by the optimization engine 40 using thesampling described herein with reference to Equations (3)-(6) entailssampling N times the translation lattice according to the conditionalprobability distribution 30 induced by the weights on its edges. Theapproach of Equations (3)-(6) uses a dynamic programming approach forcomputing the posterior probabilities of traversing edges. In this phaseeach edge of the translation lattice is visited exactly once, hence thisphase is linear in the number of edges in the lattice, hence understandard assumptions in the length J of the sentence for this case. Onceposterior probabilities are computed for the translation lattice, Npaths are sampled from it, each of which is composed of at most J edges(assuming all phrase pairs cover at least one source word). In order toselect a new edge to traverse among all possible outgoing edges from thecurrent node, the outgoing edges are suitably sorted into a binarysearch tree, storing intervals of cumulative non-normalizedprobabilities, and then a binary search is performed on it with auniformly generated random number: if |E′| is the number of successorsof a node then the computational cost is O(|E′|log|E′|) the first time anode is ever sampled (not all nodes are necessarily ever sampled) tobuild the binary tree, and then O(log|E′|) for traversing the tree foreach sampling operation. The overall cost for sampling N paths is thusO(|E|+NJ(|E′|log(|E′|)+log(|E′|))). Under standard assumptions |E′| is aconstant, and |E|≈O(J), so the whole sampling is also O(NJ), which isthe same complexity as for extracting the N-best list.

Accordingly, it is concluded that the parameter optimization techniquesdisclosed herein, which employ sampling in accordance with theconditional translation probability 30 of the available candidatetranslations (for example, as encoded in a translation lattice) hascomputational complexity that is comparable with that of the N-best listapproach. However, the sampling approaches disclosed herein providesubstantially improved performance in terms of convergence speed androbustness in terms of likelihood of converging to a set of optimizedparameters that is close to the ideal parameter values.

It will be appreciated that various of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems or applications. Also thatvarious presently unforeseen or unanticipated alternatives,modifications, variations or improvements therein may be subsequentlymade by those skilled in the art which are also intended to beencompassed by the following claims.

1. A method comprising: translating source language content in a sourcenatural language to a target natural language using statistical machinetranslation (SMT) employing a conditional translation probabilityconditioned on the source language content; and optimizing values ofparameters of the conditional translation probability by an iterativeoptimization process operating on a translation pool, the optimizingincluding adding candidate aligned translations to the translation poolby sampling available candidate aligned translations for a sourcelanguage sentence in accordance with the conditional translationprobability; wherein the conditional translation probability isquantitatively equivalent to:${P\left( {e,\left. a \middle| f \right.} \right)} = \frac{\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e,a,f} \right)}}} \right)}{\sum\limits_{{({e^{\prime},a^{\prime}})} \in L}{\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e^{\prime},a^{\prime},f} \right)}}} \right)}}$where f denotes the source language sentence, L denotes the set ofavailable candidate aligned translations, (e, a) denotes a candidatealigned translation for which the conditional probability is computed,h_(k)(. . .), k=1, . . . , K denotes a set of K feature functions, andλ_(k), k=1, . . . , K denotes the parameters of the conditionaltranslation probability; wherein the sampling comprises: selecting asampled candidate aligned translation by traversing a translationlattice representing the available candidate aligned translations forthe source language sentence from its root node to its final nodewherein each transition from a current node to a next node is selectedbased on conditional translation probabilities${\sigma\left( e_{i} \right)} = {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( e_{i} \right)}}}$of the available edges e_(i) leading away from the current node; whereinthe SMT and the optimizing are implemented by a SMT system embodied byat least one digital processor.
 2. The method as set forth in claim 1,wherein the iterative optimization process employs the bilingualevaluation understudy (BLEU) method.
 3. The method as set forth in claim1, wherein for a current iteration of the iterative optimization processthe sampling is in accordance with the conditional translationprobability employing parameter values of the current iteration.
 4. Themethod as set forth in claim 1, wherein the translating comprisescomputing:$\left( {e^{*},a^{*}} \right) = {\underset{e,a}{\arg\;\max}\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}^{opt}{h_{k}\left( {e,a,f} \right)}}} \right)}$where λ_(k) ^(opt), k=1, . . . , K denotes the parameters of theconditional translation probability optimized by the optimizing, and(e*,a*) denotes the translation.
 5. A non-transitory storage mediumstoring instructions executable on a digital processor to perform amethod including (i) translating source language content in a sourcenatural language to a target natural language based on a conditionaltranslation probability conditioned on the source language content and(ii) tuning the conditional translation probability using a translationpool, wherein the tuning includes: selecting candidate alignedtranslations for a source language sentence having a referencetranslation by sampling available candidate aligned translations for thesource language sentence in accordance with the conditional translationprobability conditioned on the source language sentence; wherein theconditional translation probability for a translation (e,a) of sourcelanguage content f is monotonically increasing with$\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e,a,f} \right)}}} \right)$ where h_(k)(. . .), k=1, . . . , K denotes a set of K feature functionsand λ_(k), k=1, . . . , K denotes tuning parameters of the conditionaltranslation probability; wherein the selecting comprises: constructing atranslation lattice representing the available candidate alignedtranslations for the source language sentence; and sampling theavailable candidate aligned translations by sampling paths through thetranslation lattice from its root node to its final node wherein eachtransition from a current node to a next node is selected based onconditional translation probabilities${\sigma\left( e_{i} \right)} = {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( e_{i} \right)}}}$ of the available edges e_(i) leading away from the current node, andadding the selected candidate aligned translations to the translationpool.
 6. The storage medium as set forth in claim 5, wherein the tuningemploys the bilingual evaluation understudy (BLEU) method.
 7. Thestorage medium as set forth in claim 5, wherein the translatingcomprises computing:$t^{*} = {\underset{t}{\arg\;\max}\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}^{opt}{h_{k}\left( {t,f} \right)}}} \right)}$where λ_(k) ^(opt), k=1, . . . , K denotes the parameters of theconditional translation probability optimized by the optimizing, and t*denotes the translation.
 8. An apparatus comprising: a statisticalmachine translation (SMT) system employing a conditional translationprobability conditioned on the source language content; and a modelparameters optimization engine configured to optimize values ofparameters of the conditional translation probability using atranslation pool comprising candidate aligned translations for sourcelanguage sentences having reference translations, the model parametersoptimization engine adding candidate aligned translations to thetranslation pool by sampling available candidate aligned translations inaccordance with the conditional translation probability; wherein theconditional translation probability is quantitatively equivalent to:${P\left( {e,\left. a \middle| f \right.} \right)} = \frac{\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e,a,f} \right)}}} \right)}{\sum\limits_{{({e^{\prime},a^{\prime}})} \in L}{\exp\left( {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( {e^{\prime},a^{\prime},f} \right)}}} \right)}}$where f denotes the source language sentence, L denotes the set ofavailable candidate aligned translations, (e, a) denotes a candidatealigned translation for which the conditional probability is computed,h_(k)(. . .), k=1, . . . , K denotes a set of K feature functions, andλ_(k), k=1, . . . , K denotes the parameters of the conditionaltranslation probability; wherein the sampling comprises: selecting asampled candidate aligned translation by traversing a translationlattice representing the available candidate aligned translations forthe source language sentence from its root node to its final nodewherein each transition from a current node to a next node is selectedbased on conditional translation probabilities${\sigma\left( e_{i} \right)} = {\sum\limits_{k = 1}^{K}{\lambda_{k}{h_{k}\left( e_{i} \right)}}}$of the available edges e_(i) leading away from the current node; whereinthe SMT system and the model parameters optimization engine are embodiedby one or more digital processors.
 9. The apparatus as set forth inclaim 8, wherein the model parameters optimization engine is configuredto iteratively optimize values of the parameters of the conditionaltranslation probability, the model parameters optimization engine ineach iteration adding candidate aligned translations to the translationpool by sampling available candidate aligned translations in accordancewith the conditional translation probability with parameter values ofthe current iteration.
 10. The method as set forth in claim 9, whereinthe iterative optimization process employs the bilingual evaluationunderstudy (BLEU) method.
 11. An apparatus comprising: A non-transitorystorage medium as set forth in claim 5; and A digital processor inoperative communication with the non-transitory computer-readablestorage medium and configured to execute instructions stored on thenon-transitory computer-readable storage medium.